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Does Small High School Reform Lift Urban Districts? 
Evidence From New York City 

Leanna Stiefel 1 , Amy Ellen Schwartz 2 , and Matthew Wiswall 3 



Research finds that small high schools deliver better outcomes than large high schools for urban students. An important 
outstanding question is whether this better performance is gained at the expense of losses elsewhere: Does small school 
reform lift the whole district? We explore New York City's small high school reform in which hundreds of new small high 
schools were built in less than a decade. We use rich individual student data on four cohorts of New York City high school 
students and estimate effects of schools on student outcomes. Our results suggest that the introduction of small schools 
improved outcomes for students in all types of schools: large, small, continuously operating, and new. Small school reform 
lifted all boats. 
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A lthough numerous policy interventions have been aimed 
at improving high school outcomes for urban students, 
“small school reform,” in which large comprehensive 
high schools are replaced by newly created small schools, is of 
particular interest for three reasons: first, because it has been 
adopted in key American cities, including New York City (NYC), 
Los Angeles, Chicago, Philadelphia, Oakland, San Diego, and 
Boston; second, because it enjoyed a substantial public and phil¬ 
anthropic funding base, including nearly $600 million each from 
the Gates Foundation and U.S. Department of Education; 1 and 
third—and perhaps most tantalizing—because recent research 
evaluating the new schools in NYC (Bloom & Unterman, 2014; 
Schwartz, Stiefel, &Wiswall [hereafter SSW], 2013) and Chicago 
(Barrow, Clessens, & Schanzenbach, 2010) suggests that stu¬ 
dents attending new small schools achieve better outcomes 
(including higher graduation rates) than students attending other 
district schools. These findings provide only part of the evidence 
needed to answer the question we pose in this paper: Does the 
introduction of new small high schools (and the corresponding 
changes in other schools) improve outcomes districtwide? 

The underlying logic of small school reform as a districtwide 
improvement strategy is threefold. First, small schools may be 
more effective than large comprehensive schools because small 
learning communities can be more intimate and nurturing and 
could attract a different mix of teachers and leaders. Second, cre¬ 
ating new small schools builds new capacity, which allows 


districts to close failing, dysfunctional schools. Third, increasing 
the number of schools means there will be more competition for 
students among schools, which could fuel innovations and 
improvements across the board. Thus, small school reform is not 
just about building better schools but about lifting all the boats. 

To be clear, the existing literature finds that students attending 
new small schools fare better than those attending old schools in 
the same period, but it does not examine changes districtwide. Is 
the better performance of new small schools gained at the expense 
of losses elsewhere? If, for example, new small schools succeed by 
immiserating the old schools—by drawing financial resources or 
high-quality students, for example—then the overall impact on 
the district’s children may well be negative, even though the stu¬ 
dents in new small schools do better. Does small school reform 
lift the whole district? This is the question we seek to address. 

In this paper, we explore the success of NYC’s small high 
school reform in which hundreds of new small high schools were 
built in less than a decade as part of a series of reforms initiated by 
NYC Public Schools chancellor Joel Klein. 2 Of particular interest 
is that the NYC Department of Education (NYCDOE) imple¬ 
mented new procedures and regulations governing new small 
schools established after 2002 (for more, see Bloom et ah, 2010; 
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Cahill & Hughes, 2010). 3 To begin, the application process 
required plans for implementing an academically rigorous cur¬ 
riculum and partnerships with community-based organizations. 
Not all applications were successful. Almost all of the new small 
schools were supported by nonprofit organizations, such as New 
Visions for Public Schools, with generous funding from the Gates 
Foundation or other philanthropies to monitor, aid, and network 
together these new small schools as they were established. Perhaps 
even more important, new small schools were granted exemp¬ 
tions in their first years from serving some groups of special needs 
students and from following all union rules on hiring teachers. 4,5 

Other key school reforms included an overhaul of the high 
school application and admissions process and changes in human 
resources policies (see Corcoran & Levin, 2011). 6 Chancellor 
Klein enjoyed strong support from mayor Michael Bloomberg, 
who was granted control over the schools by the New York State 
(NYS) Legislature in 2002. The public school budget expanded 
dramatically, fueled by favorable economic conditions in the city 
and increased state aid, and Klein served an extraordinarily long 
8-year tenure. These simultaneous changes mean that fully iso¬ 
lating the effect of small school reform from the effects of other 
changes is quite difficult. Nonetheless, in practice, reforms are 
typically implemented amid other changes—some inextricably 
linked to the reform and others merely coincident. The NYC 
initiatives offer an opportunity to gain insight into the efficacy 
of small school reform in practice and on a large scale in America’s 
largest school district, providing implications for policymakers 
and education reform leaders. 

In this paper, we use a rich administrative data set of indi¬ 
vidual student data on four cohorts of NYC high school stu¬ 
dents; two were slated to graduate from high school prior to 
Bloomberg’s takeover of the schools (2001 and 2002) and two 
were expected to graduate after the small school reforms were 
well underway (2007 and 2008). We estimate a model of school 
outcomes controlling for student characteristics to examine 
changes over time and explore the differential effects of small 
schools. We control for potential selection into small schools 
using an instrumental variables (IV) approach as in SSW (2013). 
In addition, we explore differences between schools that were 
closed, new schools that opened, and importantly, the gains 
made in the continuously operating schools. Did high school 
outcomes improve across the board? To what extent are observed 
changes reflective of changes in the student body? Did old 
schools improve or were gains driven by replacing low-perform¬ 
ing schools with high-performing schools? We hope to provide a 
nuanced picture of the effects of the small school reform overall. 
Our evaluation, therefore, is relevant for policymakers who aim 
to initiate small high school reform in an environment—like 
that found in many urban school districts in the United States 
today—where change and reform is ongoing and “business as 
usual” involves continual change. 

We begin by reviewing the key literature on small schools. 
Next, we turn to describing our data, and in the following sec¬ 
tion, we explain our models. The fifth section presents results on 
the overall impact of school reforms, followed by an exploration 
of the gains made overall and by small schools in particular. The 
final section concludes with the implications for policymakers 
and education reformers. 
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What Do We Know About Small Schools? 

Much of the existing literature on small schools is correlational 
and microfocused—aimed at understanding how small high 
schools differ from large high schools or how outcomes vary with 
school size. Fowler and Walberg (1991), Fowler (1992), and Lee 
and Smith (1997), for example, find that achievement scores and 
attendance rates are higher and dropout rates are lower in small 
schools compared to large schools. Fowler (1992) and Page, 
Layzer, Schimmenti, Bernstein, and Horst (2002) suggest that 
small schools have more student participation in extracurricular 
activities and better student and teacher attitudes. Shouse (2004) 
and Powell, Farrar, and Cohen (1985), among others, find stu¬ 
dents in large schools have less personal relationships with teachers 
and more student disengagement due to feelings of anonymity. 

Lee and Smith (1997) go further to address the question of 
what size enrollment is “small.” They report that an optimal 
school size with respect to maximizing student achievement 
ranges between 600 and 900 students, which is larger than that 
promoted in most current initiatives, including NYC’s. 7 

Although valuable, this research does not offer evidence on 
the causal relationship between school size and achievement. 
Schneider, Wyse, and Keesler (2007), among the first to explic¬ 
itly address the issue of causality, use hierarchical linear modeling 
(HLM) and propensity score matching to attempt to control for 
selection into small schools. Although the HLM results suggest 
attending a small high school has little effect on achievement, 
postsecondary expectations, and number and types of college 
applications, propensity score matching results suggest some¬ 
what more positive impacts of small schools. 

More recently, three studies have made important strides in 
obtaining causal estimates of small school efficacy. Barrow 
et al. (2010) and SSW (2013) use distance between residence 
and school choices as an IV to address potential endogeneity in 
the choice to attend a small high school in Chicago and NYC, 
respectively. Bloom and Unterman (2014) exploit a lottery 
design to examine the outcomes of randomly assigned lottery 
winners and losers at oversubscribed small high schools in 
NYC. 

More specifically, SSW (2013) evaluate the impact of small 
high schools (enrolling 550 or fewer students) on all first-time 
ninth graders in two NYC cohorts. They find that 121 new 
small schools (graduating classes after 2002) delivered higher 
graduation outcomes (17.5 percentage points higher) for attend¬ 
ing students in 2007 and 2008, compared to 122 large schools 
operating in that year. Further, they find that the 48 old small 
schools (graduating classes 2002 and before) had significantly 
worse outcomes, with 56 percentage points lower graduation 
rate relative to large schools. These results are based on IV regres¬ 
sions and include rich controls for student sociodemographic 
characteristics. Bloom and Unterman (2014) examine 105 over¬ 
subscribed small high schools that held lotteries to allocate places 
in the school and find positive effects of small schools on persis¬ 
tence through high school but no improvement on test scores. 
Specifically, comparing the outcomes of lottery winners and los¬ 
ers for one cohort of students, they find the 4-year graduation 
rates for students attending small high schools are 6.8 percentage 
points higher than those of the students in the control group. 



To date, however, there is no evidence, whether descriptive or 
causal, on whether districts adopting small school reform 
improve overall. Current literature focuses entirely on whether a 
district’s small schools are better than large schools operating 
concurrently. There is very little evidence on whether the large 
schools suffer in the face of increased attention on their small 
school counterparts or whether the new small schools replace 
bad old schools. In this paper, we take a longer perspective by 
examining changes in high school outcomes over the course of 
the reform years. Further, we extend the definition of success in 
the small schools reform movement by focusing attention not on 
whether small schools are high performing but on whether all 
schools—small and large—improved performance on key high 
school outcomes during this time period. 

Data 

We use richly detailed student-level administrative data from the 
NYCDOE for the four cohorts of public high school students 
expected to graduate in 2001, 2002, 2007, and 2008. 
Throughout the paper, we regard cohorts 2001 and 2002 as pre¬ 
reform and cohorts 2007 and 2008 as postreform. These student- 
level data include information on the student’s gender, race/ 
ethnicity, poverty (measured as participation in the free lunch 
program in eighth grade), English proficiency, home language, 
whether the student is overage for grade, and performance on 
standardized English language arts (ELA) and math exams. 8 
Additionally, we have data on whether students graduated in 4 
years and on their test taking and performance on statewide 
English and Math Regents exams. 9 We define graduation as earn¬ 
ing a local, Regents, Honors, or Advanced Regents diploma in 
four years. 1 " Finally, we have data on the student’s residence bor¬ 
ough and residence zip code, which we use to calculate distances 
between students’ homes and the nearest schools. 

We assign each student to his or her ninth-grade school fol¬ 
lowing an “intent-to-treat” strategy. We use geocoded high 
school addresses and enrollment information from the New York 
State Department of Education’s (various years) School Report 
Cards and the NYCDOE’s (various years) School Based 
Expenditure Reports. Across the literature on school size and out¬ 
comes, there is no universal agreement on a definition of small. 
The federal government, through its Small Schools Initiative, set 
a limit of 300 students; the Gates-funded initiative in NYC con¬ 
sidered 500 students the upper limit for small high schools; pre¬ 
vious research on the costs of small high schools in NYC, as well 
as the then-current local policy, considered 600 students or fewer 
small (Stiefel, Berne, Iatarola, & Fruchter, 2000); and Lee and 
Smith (1997) found schools in the range of 600 to 900 to be 
most effective for minority students. Recent work in NYC 
defines small as enrolling 550 or fewer students (Bloom & 
Unterman, 2014; SSW, 2013), and to remain consistent with 
these recent studies, we use this definition in our analysis. 

We exclude students attending alternative high schools (such 
as “last-chance” high schools or schools for pregnant mothers), 
schools designed to serve only special education students, and 
charter schools. Charter high schools are rare in NYC, and data 
are not available on the ones that exist. We also exclude schools 
and students in Staten Island. 11 


Empirical Methods 

We begin by estimating a regression model linking student out¬ 
comes to a set of student sociodemographic and educational 
characteristics and a set of cohort fixed effects: 

HS_outcome i[W = a + X ; .[3 + 0 A + 5^, + e M . (1) 

Here HS_outcome is a student outcome for student i in school 
s, in cohort k, residing in borough b} 1 We explore five main 
outcomes of interest: graduation rates, English and Math 
Regents test-taking rates, and passing rates at or above 65 on the 
English and Math Regents. In our specifications, we include a 
vector of student characteristics (X), including a set of eighth- 
grade ELA and math test scores (each score, each score squared, 
and scores interacted) as well as indicators for eligibility for free 
or reduced-price lunch, gender, race, and so on. We include the 
square and interacted scores to capture any possible nonlineari¬ 
ties in the relationship between the outcome and the prior test 
scores, and these were statistically significant. In general, when 
the sample is large (as ours is) and thus there is no problem with 
degrees of freedom, including such a set of covariates is helpful 
for controlling for possible sources of omitted variables. We also 
include borough fixed effects (Q b ) and cohort fixed effects (5^) to 
allow for changes in the outcome of interest over time. For all 
specifications, we compute heteroskedastic robust standard 
errors that are clustered at the school level. 

We then extend this model to include an indicator variable 
taking a value of 1 if student i in cohort k attended a small school 
in his or her ninth-grade year and 0 otherwise. Here, small is 
defined as enrolling 550 or fewer students. We interact this vari¬ 
able with cohort indicators, allowing us to estimate cohort-specific 
coefficients on the small schools indicator. 

HS_outcome liW = a + X.(3 + 0^+5^+ SM rf + s M , (2) 

where SM^ is a series of indicators for small school interacted 
with each cohort: 2001, 2002, 2007, and 2008. In this specifica¬ 
tion, 5^ provides the effect of attending a large school for each 
cohort (compared to the 2001 cohort, the omitted cohort group) 
and SM (/ , provides the effect of attending a small school relative 
to a large school for each cohort. 

Finally, we address the potential bias that might arise if stu¬ 
dent selection into small schools is driven by variables unobserved 
in our data set. To do so, we use an IV approach. In this approach, 
a set of IVs is used to predict whether a student attends a small 
high school (the treatment or endogenous variable), and then this 
prediction for each student is used to estimate the effect of small 
high school attendance on outcomes. The prediction, in effect, is 
“substituted” for the small school indicator to avoid the issue of 
selection into the schools, based on the logic that the IVs are 
related only to attendance and not to outcomes except through 
their effect on attendance. By instrumenting for small school 
attendance at each cohort, we are also implicitly instrumenting 
for large school attendance since large school attendance is the 
residual category (as represented by the cohort effects, 5^). 

Two assumptions underlie this approach to addressing selec¬ 
tion bias. The IV must predict the actual treatment the student 
received, small or large school, and the IV cannot influence the 
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outcome variable through any other channel but school atten¬ 
dance. This latter assumption implies that there is no direct rela¬ 
tionship between the IV and outcomes, and there must be no 
common determinants of the IV and the outcome. 

For our IVs, we use the distance between a student’s residence 
and the nearest small and large school. 13 To be specific, following 
SSW (2013), we calculate the minimum Euclidean distance 
from the centroid of each residence zip code to geocoded 
addresses of small and large schools. 14 We include the minimum 
distance to small schools, its square, the minimum distance to 
large schools, and its square. Distances are calculated using the 
students’ eighth-grade residence zip code. The intuition is that 
the likelihood of attending any particular school decreases as the 
distance to the school increases, reflecting higher transportation 
costs broadly defined, information costs, and so on. The coeffi¬ 
cients on the variables measuring distance from a small school 
and its square are statistically significant and plausible, and the F 
statistic for the total regression or for the excluded instruments is 
large, indicating that our distance instruments provide strong 
instruments for small school attendance (Staiger & Stock, 1997). 

Although the second assumption for the IV to be appropriate 
cannot be tested empirically, SSW (2013) provide evidence for 
the validity of the instrument. In that work, various threats to 
validity of the instrument, including selective location of small 
schools, student mobility in response to school locations, and 
student entry or exit into the NYC public school system, were 
tested and found not to be important areas for concern. The 
various tests and additional evidence in SSW (2013) indicates 
that the IV estimates would provide consistent estimates of the 
effect of schools on student outcomes. 

The main limitation to the IVs approach is that the IV esti¬ 
mates only a “local” effect, where local is specific to the “compli- 
ers” to the particular instrument—that is, those for whom 
enrollment in a small high school depends upon the IV used 
(Angrist & Imbens, 1994). With our distance-based instru¬ 
ments, then, the compliers are students who are sensitive to 
school distance. The effect we estimate is then specific to this 
subpopulation and may not represent the average effect in the 
population of all NYC public school students. 

Results 

We begin by comparing the characteristics and performance of 
NYC high schools and high school students in two prereform 
cohorts—2001 and 2002—and two postreform cohorts—2007 
and 2008. We next consider the regression results, organized 
around answering four questions: Are high school outcomes 
improving? Is the apparent improvement reflecting student pop¬ 
ulation changes? Is the improvement limited to small schools, or 
are all rising? Are the results robust to selection? 

NYC High Schools and Students by the Numbers 

As shown in Table 1, the small school reform significantly 
changed the portfolio of schools. In the 2001 cohort, there were 
only 60 small schools serving roughly 9% of the city’s first-time 
ninth graders. In 2007, the number of small schools had 
increased over 92%: Roughly 19% of the city’s first-time ninth 
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graders attended over 100 small schools. The number of small 
schools continued to grow—by 41 schools between cohorts 
2007 and 2008. Nonetheless, the vast majority of the city’s 
ninth-grade students (78%) attended large schools in 2008. 

As new small schools opened, the average minimum distance 
between student residences and small high schools decreased. 
Consistent with the notion that distance matters, the change was 
larger for students attending small schools (roughly 1 mile in the 
prereform cohorts to 0.7 to 0.8 miles in the postreform cohorts) 
than for students attending large schools, but the distance to the 
nearest small school decreased for students attending large 
schools as well. At the same time, distance to the nearest large 
high school remained relatively constant, roughly 0.6 miles, 
throughout the time period, and on average, students in all 
cohorts had a large school as the nearest one. 

This period also saw changes in the characteristics of students 
served by NYC public high schools. Most notably, the percent¬ 
age of Black and White students declined, and the percentage of 
Hispanic and Asian students increased. The proportion of stu¬ 
dents who spoke English at home declined, as did the percentage 
overage for grade. 

Small schools serve somewhat different students than large 
schools. In all years, students in small high schools had lower 
performance on their eighth-grade exams than students in non¬ 
small high schools, although the differential with large schools 
declined over this period. Students who attend small high 
schools were more likely to be eligible for free lunch, Black or 
Hispanic, female, and overage for grade. 15 

In summary, although the prevalence of small schools 
increased significantly during this time period, the majority of 
ninth-grade students in cohort 2008 is still served by, and reside 
near, large schools. In each cohort, the composition of students in 
small and large high schools differs, with small high schools gen¬ 
erally serving less advantaged and lower-achieving populations. 

Are High School Outcomes Improving? 

In order for small schools reform to work as systemic reform, 
performance overall citywide needs to increase. We begin to 
examine this question by analyzing the unadjusted (or raw) 
changes in the five high school outcomes. As shown in Table 2, 
graduation rates rose over time, increasing 16.8 percentage points 
between 2001 and 2008. Although only 51% of NYC students 
graduated high school in 4 years in 2001, 64% graduated in 4 
years in 2007, and 68% graduated in 4 years in 2008. At the same 
time, we see similar increases over the period in test taking and 
scores. Roughly 75% of students took each exam in 2001; by 
2008, over 85% of students took the English Regents, and nearly 
86% took the Math Regents. The share passing with a score at or 
above 65 increased roughly 19 percentage points on both exams 
between 2001 and 2008. In sum, NYC’s students, as a whole, 
improved on all high school outcomes during this period. 

Is the Apparent Improvement Reflecting Student 
Population Changes? 

As noted earlier, NYC saw changes in its student body that may 
explain the rise in performance, separate and apart from the 
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Table 2 


Unadjusted Regression Results, Baseline Models, All Schools 


Variable 

(D 

(2) 

(3) 

(4) 

(5) 

Graduated 

Took English 
Regents 

Took First Math 
Regents 

English Regents 
Score > 65 

Math Regents 
Score > 65 

2002 

0.016*** 

0.016*** 

-0.014*** 

0.067*** 

-0.012*** 


(0.004) 

(0.003) 

(0.003) 

(0.004) 

(0.004) 

2007 

0.129*** 

0.071*** 

0.058*** 

0.176*** 

0.185*** 


(0.004) 

(0.003) 

(0.003) 

(0.004) 

(0.004) 

2008 

0.168*** 

0.113*** 

0.106*** 

0.188*** 

0.189*** 


(0.004) 

(0.003) 

(0.003) 

(0.004) 

(0.004) 

Constant 

0.509*** 

0.740*** 

0.752*** 

0.683*** 

0.651*** 


(0.003) 

(0.002) 

(0.002) 

(0.003) 

(0.003) 

Student controls 

N 

N 

N 

N 

N 

Number of schools 

293 

293 

293 

292 

292 

Observations 

138,215 

138,215 

138,215 

109,826 

109,758 

FP 

.021 

.012 

.014 

.037 

.049 


Note. Robust standard errors, adjusted for within-school clusters, in parentheses. The 2001 cohort is the omitted group, and its outcomes are indicated by the constant. The 
other year coefficients indicate differences from the total graduation rate in 2001. N = no. 

***p < .01. **p < .05. *p < .1. 


reforms. Since the unadjusted results presented in Table 2 do not 
control for any student characteristics, the estimates may reflect 
changing populations, not improvements driven by small school 
reform. 

To explore this, we estimate the model controlling for a range 
of student characteristics. As shown in Table 3, the results are 
remarkably unchanged. The 4-year cohort graduation rate 
increased 16.3 percentage points from a base of 52.6% in 2001 
to roughly 66% in 2008. There are similar improvements for 
Regents test taking and passing: On both exams, the share of 
students taking the test increased 10 percentage points to 
approximately 88%, and the share passing with a 65 or above 
increased 20 percentage points on both exams. Again, evidence 
suggests that performance on key high school outcomes 
improved during these 7 years, taking into account the composi¬ 
tion of students. 

Is the Improvement Limited to Small Schools, or Did 
the Large Schools Improve as Well? 

As seen in Table 4, even when we allow for differential effects for 
the small schools, there are still significant gains across the board: 
Graduation rates for large schools are 14 percentage points 
higher in 2008 compared to 2001. Compared to their peers in 
large schools, however, students attending small high schools 
have higher graduation rates in all cohorts, and the differential 
increased by over 5 percentage points (for 2001 to 2008). In the 
postreform years (2007 and 2008), students attending small 
schools are 10% to 13% more likely to graduate in 4 years than 
their otherwise similar peers in large schools. 

Turning to Regents exams, students attending large schools 
in 2008 were 8 to 10 percentage points more likely to take the 
exams, and passing rates at or above 65 were 20 percentage 
points higher compared to 2001. Students in small schools in 
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the postreform cohorts are 7 to 8 percentage points more likely 
to take these exams relative to their otherwise similar peers 
attending large schools. Students attending small schools in 
2008 are slightly more likely to pass with a 65 on the Math 
Regents compared to students attending large schools. Their 
passing rates on the English Regents, however, are not signifi¬ 
cantly different from those of the students attending large 
schools in the postreform cohorts. 

The gap in passing rates between the small and large schools 
decreased over the time period. In the prereform cohorts, stu¬ 
dents attending small schools were roughly 6 percentage points 
less likely to pass the English Regents with a 65; by 2008, they 
were no less likely to pass compared to their large school peers. 
On the Math Regents in 2002, students in small schools are less 
likely to pass, but by 2008, they are slightly more likely to pass. 

Overall, allowing for differential effects for the small schools 
does not change the finding that there were improvements in 
graduation rates and Regents test taking and passing during this 
time period for the large schools. 

Are the Results Robust to Selection? 

Table 5 presents the IV estimates for the effect of small school 
attendance for our five main outcomes of interest. In this specifi¬ 
cation, we replicate Table 4 but using IV estimation and allowing 
coefficients on the covariates to differ by early and late cohorts. 16 

Importantly, the IV estimates differ dramatically from the 
ordinary least squares estimates. There are still significant and 
large improvements for the students attending large schools— 
and, actually, there are significant and large improvements for 
students in small schools. But here we find that in the early 
cohorts, students in small schools are less likely to earn a diploma 
in 4 years. In the postreform years, graduation rates are similar 
for students in small and large schools. 17 





Table 3 

Adjusted Regression Results, Baseline Models, All Schools 


Variable 

(D 

(2) 

(3) 

(4) 

(5) 

Graduated 

Took English 
Regents 

Took First Math 
Regents 

English Regents 
Score > 65 

Math Regents 
Score > 65 

2002 

0.015*** 

0.015*** 

-0.015*** 

0.066*** 

-0.018*** 


(0.003) 

(0.003) 

(0.003) 

(0.004) 

(0.004) 

2007 

0.121*** 

0.062*** 

0.050*** 

0.184*** 

0.194*** 


(0.003) 

(0.003) 

(0.003) 

(0.003) 

(0.003) 

2008 

0.163*** 

0.108*** 

0.102*** 

0.204*** 

0.209*** 


(0.003) 

(0.003) 

(0.003) 

(0.003) 

(0.003) 

Constant 

0.526*** 

0.775*** 

0.782*** 

0.669*** 

0.648*** 


(0.005) 

(0.004) 

(0.004) 

(0.005) 

(0.005) 

Student controls 

Y 

Y 

Y 

Y 

Y 

Number of schools 

293 

293 

293 

292 

292 

Observations 

138,215 

138,215 

138,215 

109,826 

109,758 

FP 

.254 

.153 

.163 

.263 

.279 

Note. Robust standard errors, adjusted for within-school clusters, in parentheses. The 2001 cohort is the omitted group, and its outcomes are indicated by the constant. 

The other year coefficients indicate differences from the total graduation rate in 2001. 

All models include controls for gender, race/ethnicity, overage for grade, English 

proficiency, home language, poverty (measured as eligibility for free lunch), eighth-grade test scores on standardized English language arts and math exams, and residence 

borough. Y = yes. 






***p < .01. **p < .05. *p < .1 








Table 4 





Adjusted Regression Results, All Schools 



(D 

(2) 

(3) 

(4) 

(5) 



Took English 

Took First Math 

English Regents 

Math Regents 

Variable 

Graduated 

Regents 

Regents 

Score > 65 

Score > 65 

2002 (large) 

0.017*** 

0.015*** 

-0.017*** 

0.066*** 

-0.013** 


(0.004) 

(0.004) 

(0.007) 

(0.007) 

(0.006) 

2007 (large) 

0.108*** 

0.054*** 

0.029*** 

0.180*** 

0.195*** 


(0.010) 

(0.008) 

(0.010) 

(0.015) 

(0.015) 

2008 (large) 

0.140*** 

0.097*** 

0.078*** 

0.197*** 

0.201*** 


(0.010) 

(0.008) 

(0.012) 

(0.016) 

(0.016) 

Small vs. large 






2001 

0.079*** 

0.040* 

-0.073* 

-0.055** 

-0.031 


(0.016) 

(0.021) 

(0.039) 

(0.027) 

(0.028) 

2002 

0.049** 

0.040*** 

-0.053 

-0.062*** 

-0.092*** 


(0.020) 

(0.014) 

(0.039) 

(0.020) 

(0.029) 

2007 

0.107*** 

0.066*** 

0.076*** 

-0.010 

-0.015 


(0.013) 

(0.009) 

(0.010) 

(0.011) 

(0.014) 

2008 

0.133*** 

0.067*** 

0.080*** 

0.011 

0.025** 


(0.012) 

(0.010) 

(0.013) 

(0.011) 

(0.012) 

Constant 

0.509*** 

0.765*** 

0.783*** 

0.674*** 

0.650*** 


(0.013) 

(0.011) 

(0.015) 

(0.016) 

(0.015) 

Student controls 

Y 

Y 

Y 

Y 

Y 

Number of schools 

293 

293 

293 

292 

292 

Observations 

138,215 

138,215 

138,215 

109,826 

109,758 

FP 

.260 

.156 

.167 

.264 

.280 


Note. Robust standard errors, adjusted for within-school clusters, in parentheses. Small schools are those with 550 or fewer students in that cohort year. All models include 
controls for gender, race/ethnicity, overage for grade, English proficiency, home language, poverty (measured as eligibility for free lunch), eighth-grade test scores on 
standardized ELA and math exams, and residence borough. The 2001 cohort attending large schools forms the omitted group, and its outcomes are indicated by the constant 
for the group of students defined by the student covariates. The other year coefficients indicate differences from the 2001 graduation rate for students attending large schools. 
The small cohort coefficients capture the difference between the outcomes for students attending small compared to large schools in that cohort year. Y = yes. 

***p < .01. **p < .05. *p < .1. 
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Table 5 


Instrumental Variable Regression Results, All Schools 



(D 

(2) 

(3) 

(4) 

(5) 

Variable 

Graduated 

Took English 
Regents 

Took First Math 
Regents 

English Regents 
Score > 65 

Math Regents 
Score > 65 

2002 (large) 

0.012 

0.017** 

-0.014 

0.062*** 

-0.003 


(0.009) 

(0.008) 

(0.010) 

(0.010) 

(0.012) 

2007 (large) 

0.195*** 

0.069*** 

0.056** 

0.329*** 

0.287*** 


(0.032) 

(0.024) 

(0.027) 

(0.032) 

(0.029) 

2008 (large) 

0.224*** 

0.101*** 

0.090*** 

0.333*** 

0.294*** 

Small vs. large 

(0.033) 

(0.023) 

(0.027) 

(0.032) 

(0.029) 

2001 

-0.431** 

-0.247* 

-0.227 

-0.350** 

-0.488*** 


(0.174) 

(0.129) 

(0.154) 

(0.142) 

(0.188) 

2002 

-0.427** 

-0.294** 

-0.247 

-0.322** 

-0.699*** 


(0.171) 

(0.132) 

(0.167) 

(0.142) 

(0.223) 

2007 

-0.022 

0.006 

0.024 

-0.285*** 

-0.144*** 


(0.067) 

(0.055) 

(0.062) 

(0.060) 

(0.053) 

2008 

0.043 

0.067* 

0.099** 

-0.180*** 

-0.098** 


(0.056) 

(0.037) 

(0.044) 

(0.044) 

(0.042) 

Constant 

0.506*** 

0.778*** 

0.781*** 

0.650*** 

0.635*** 


(0.026) 

(0.018) 

(0.024) 

(0.023) 

(0.025) 

Student controls 

Y 

Y 

Y 

Y 

Y 

Observations 

138,215 

138,215 

138,215 

109,826 

109,758 

FP 

.224 

.137 

.165 

.233 

.239 


Note. Robust standard errors, adjusted for within-school clusters, in parentheses. Small schools are those with 550 or fewer students in that cohort year. All models 
include controls for gender, race/ethnicity, overage for grade, English proficiency, home language, poverty (measured as eligibility for free lunch), eighth-grade test scores 
on standardized English language arts and math exams, and residence borough. All covariates for the specification in column 5 are also interacted with an indicator for 
late (2007 and 2008) cohorts. The 2001 cohort attending large schools forms the omitted group, and its outcomes are indicated by the constant for the group of students 
defined by the student covariates. The other year coefficients indicate differences from the 2001 graduation rate for students attending large schools. The small cohort 
coefficients capture the difference between the outcome for students attending small compared to large schools in that cohort year. Y = yes. 

***p < .01. **p < .05. *p < .1. 


We see similar results in Regents outcomes: Students attending 
small schools in the prereform years are less likely to take the 
English Regents and less likely to pass either the English or math 
examination. In 2008, students attending small schools are between 
7 percentage points and 10 percentage points more likely to take 
the English or math exam. Performance on both the English and 
math exams remains significantly worse for students attending 
small schools compared to large schools in the postreform cohort 
years but better than in small schools in earlier cohorts. 

Probing the Results 

Our results suggest that there was real, meaningful improvement 
in high school outcomes during this time period. A significant 
component of small school reform in NYC is changing the com¬ 
position of schools by opening new schools and closing bad 
schools. Fewer than half of the 293 schools operating in at least 
one of these cohort years operated continuously though the 
period; 20 schools that operated in 2001 were no longer operat¬ 
ing in 2008, and over 120 schools opened. 18 At the same time, 
the 144 schools operating continuously throughout this period 
served a significant portion of the first-time ninth graders: four 
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in five students in cohort 2008 attend a school that operated 
continuously throughout this period. Did these schools improve 
as well, driven perhaps by competition, or did they languish? 

Continuously Operating Schools 

To address this, we replicate our main analyses restricting the 
sample to the 144 schools operating in all 4 cohort years. In 
these models, we include school fixed effects, so the estimated 
gains over time capture gains made within schools and not 
changes in the mix of schools, that is, between schools. 

As seen in Table 6, performance improved on all outcomes for 
the 144 continuously operating schools. Students attending large 
continuously operating schools in the postreform cohorts had 
improvements in graduation rates of approximately 8 to 10 per¬ 
centage points. In 2008, students attending small continuously 
operating schools had an additional increase in their graduation 
rates of approximately 5 percentage points. Regents test-taking 
rates improved 6 to 7 percentage points for the large schools, and 
passing rates improved 18 percentage points. Small continuously 
operating schools had significantly higher shares of students tak¬ 
ing the Math Regents and passing the English and Math Regents. 





Table 6 

Ordinary Least Squares Regression Results, Continuously Operating Schools 



(D 

(2) 

(3) 

(4) 

(5) 

Variable 

Graduated 

Took English 
Regents 

Took First Math 
Regents 

English Regents 
Score > 65 

Math Regents 
Score > 65 

2002 (large) 

0.012** 

0.011** 

-0.016** 

0.059*** 

-0.016*** 


(0.005) 

(0.004) 

(0.007) 

(0.007) 

(0.006) 

2007 (large) 

0.083*** 

0.037*** 

0.016 

0.166*** 

0.176*** 


(0.009) 

(0.007) 

(0.010) 

(0.015) 

(0.014) 

2008 (large) 

0.105*** 

0.074*** 

0.057*** 

0.180*** 

0.179*** 

Small vs. large 

(0.009) 

(0.007) 

(0.011) 

(0.016) 

(0.015) 

2001 

— 

— 

— 

— 

— 


— 

— 

— 

— 

— 

2002 

-0.009 

0.004 

-0.007 

0.037 

-0.030 


(0.024) 

(0.027) 

(0.029) 

(0.029) 

(0.028) 

2007 

0.036 

0.056 

0.251*** 

0.119*** 

0.096*** 


(0.025) 

(0.035) 

(0.055) 

(0.036) 

(0.030) 

2008 

0.054** 

0.055 

0.240*** 

0.119*** 

0.144*** 


(0.024) 

(0.034) 

(0.055) 

(0.037) 

(0.034) 

Constant 

0.534*** 

0.787*** 

0.785*** 

0.693*** 

0.669*** 


(0.011) 

(0.008) 

(0.009) 

(0.014) 

(0.013) 

Student controls 

Y 

Y 

Y 

Y 

Y 

School FX 

Y 

Y 

Y 

Y 

Y 

Number of schools 

144 

144 

144 

144 

144 

Observations 

117,380 

117,380 

117,380 

94,101 

93,704 

FF 

0.266 

0.161 

0.197 

0.279 

0.295 


Note. Robust standard errors, adjusted for within-school clusters, in parentheses. Continuously operating small schools are those with 550 or fewer students in all cohort 
years (and therefore are can be thought of as “always small”). All models include controls for gender, race/ethnicity, overage for grade, English proficiency, home language, 
poverty (measured as eligibility for free lunch), eighth-grade test scores on standardized English language arts and math exams, and residence borough. The 2001 cohort is 
the omitted group, and its outcomes are indicated by the constant for the group of students defined by the student covariates. Y = yes; FX = fixed effects. 

***p < .01. **p < .05. *p < .1. 


Schools That Closed and Opened 

Finally, we examine the performance of the schools that closed 
and those that opened. To do so, we examine the regression- 
adjusted differences for these three school types in each cohort. 
For ease of interpretation, the regression-adjusted differences for 
graduation rates are presented in Figure 1. In this figure, we do 
not distinguish between small and large schools in each category 
because none of the schools that opened in this period were 
large, and almost none of the closed schools were small. We dis¬ 
play differences between large and small schools that continu¬ 
ously operated in the previous section. IV results for our five 
main high school outcomes are available from the author. 

As seen in Figure 1, continuously operating schools improved 
their graduation rates across the four cohorts. Schools that closed 
had significantly lower graduation rates: 12.4 percentage points 
lower than continuously operating schools in 2001. Schools that 
were closed still performed below the continuously operating 
schools in 2002, although the differential in their performance was 
slightly smaller than that for the 2001 cohort. Schools closing in 
2007 had only slightly lower graduation rates compared to continu¬ 
ously operating ones that year. Overall, the schools that closed had 



2001 2002 2007 2008 

■ continuously operating ■ closed Inew 


FIGURE 1. Graduation rates, continuously operating, closed, and 
new schools, all cohorts 

lower graduation rates compared to the continuously operating 
schools in each cohort. The differential decreased over time, sug¬ 
gesting that the district closed the “worst of the worst” schools first. 

We find new schools have significantly higher graduation 
rates than continuously operating schools in the 2007 and 2008 
cohorts. Students attending new schools have graduation rates 
that are roughly 13 percentage points higher than their peers in 
continuously operating schools in the postreform cohorts. 
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Conclusions 

Did high school outcomes improve in NYC as the small school 
reform was implemented? Our results suggest that they did: 
Graduation, Regents test taking, and Regents passing rates have 
all improved significantly since 2001. Moreover, these improve¬ 
ments occurred in the large schools, in the small schools, in the 
continuously operating schools, and in the new schools. Thus, 
there is some support for the notion that small school reform 
works as systemic reform. 

Of course, there were many other changes occurring—both 
in NYC and in NYS—and as noted by Kemple (2011), 

some amount of this improvement is likely an artifact of reforms 
and trends that were under way .... some is likely due to other 
reform initiatives at the federal and state level, and some is likely 
due to a growing familiarity with the assessments and testing 
strategies, (p. 288) 

At the same time, in this period, the unemployment rate in NYC 
decreased from 5.8% to 5.4%, whereas unemployment state¬ 
wide rose from 4.7% to 5.3%. 19 Thus, economic conditions in 
NYC were improving absolutely and compared to the state. 

Even more important, as reported in Stiefel and Schwartz 
(2011), spending on education increased dramatically in this 
period. Per-pupil revenues increased over $5,000 in inflation- 
adjusted dollars between 2002 and 2008 compared to an increase 
of $3,200 in the rest of the state; 58% of this growth was due to 
a $3,400 increase in local dollars. 20 Prereform, NYC spent less 
per pupil than the rest of the state, but postreform (in 2008), the 
city spent roughly $1,500 more per pupil. Between 2002 and 
2008, NYC increased per-pupil total expenditures almost $4,400 
in inflation-adjusted dollars. 21 

Additionally, these years witnessed significant changes in 
practices surrounding teachers. NYS regulations required that as 
of 2003, all newly hired teachers had to be certified. This, in 
combination with the growth of alternative certification pro¬ 
grams, such as the NYC Teaching Fellows and Teach for America, 
meant the characteristics of the teaching workforce in the 2000s 
were different from those in the 1990s. Moreover, Bloomberg 
and Kleins push to increase principal authority in exchange for 
increased school accountability meant that as the decade went 
on, principals had tools and systems, such as the open market 
transfer system, that theoretically allowed them to “match the 
needs of their students and schools to the characteristics of 
teachers”(Goertz, Loeb, & Wyckoff, 2011, p. 174). 

Were the gains in NYC merely a reflection of statewide 
reforms or macro effects? Kemple (2011) examines student out¬ 
comes in NYC and NYS between 2003 and 2010. He finds 
NYC increased performance on fourth- and eighth-grade ELA 
and math proficiency rates and graduation rates. More impor¬ 
tantly, regression-adjusted estimates provide evidence the city 
pulled away from the rest of state on all of these measures during 
this time period. 22 Thus, it seems likely that the gains in high 
school outcomes were, at least in part, reflections of changes 
affecting all students and not just high school students, and in 
particular, ones that affected earlier grades may have been carried 
into high school as the students aged. 
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Bringing Small School Reform to Scale 

What would it mean to bring small school reform to scale? It 
seems implausible to eliminate all large high schools and replace 
them with small schools in large urban districts. Based on ninth- 
grade enrollments from the National Center for Education 
Statistics Common Core of Data for the four largest school dis¬ 
tricts, educating all high school students in small schools would 
require 640 small schools in NYC, 511 in Los Angeles, 256 in 
Chicago, and 190 in Miami. 23 These are very large numbers of 
schools to manage, and given that they also cost more per pupil 
(see SSW, 2013; Stiefel, Schwartz, Iatarola, & Chellman, 2009), 
they would be expensive. 

Instead, one can imagine creating new small schools incre¬ 
mentally. If adding new small schools improves outcomes in 
large schools as well as small, it is possible that the benefits of 
adding more schools decreases, and ultimately, there is an effi¬ 
cient portfolio of small and large schools that takes into account 
the benefits as well as the costs of running small high schools. 

In the end, districts intending to use the creation of small 
high schools as a systemic reform need to be aware that this strat¬ 
egy involves closing poorly performing large schools and setting 
up a process to govern how new small schools will be established. 
Moreover, given the higher costs of small schools and the saluta¬ 
tory effects on continuously operating large schools, the reform’s 
goals could be to establish a portfolio of similarly performing 
large and small schools. That is, small schools could be created 
and large poorly performing ones closed until the performance 
of large and small schools converge. 

NOTES 

This research benefited from support from the Institute for 
Education Sciences, U.S. Department of Education, Grant Number 
R305A080552. All analyses and conclusions are the responsibility of 
the authors, however. 

'Please see the literature review for evidence on this and other 
statements in this section. 

2 Over the past decade, the Gates Foundation has invested over 
$700 million for high school initiatives, including $590 million (80%) 
on reforms in which small schools are either the centerpiece or an essen¬ 
tial component of the reform (i.e., early-college high school programs). 
Likewise, the U.S. Department ofEducation awarded grants totaling $140 
million as part of its Smaller Learning Communities initiative from 2001 
to 2002, with an additional $477 million appropriated for 2002 to 2004. 

3 The investment in new small high schools continues, along with 
continued investment in new charter schools and new middle schools. 
In his 2012 State of the City, Mayor Bloomberg (2012) reported, 

The four new schools here at the Morris campus are among the 
500 new schools we’ve created over the past decade, including 
139 new charter schools. This year, we’ll phase out another 25 
schools and open smaller schools in the same buildings. All told, 
our goal is to open 100 new schools over the next two years— 
including 50 new charters. 

4 Some of these exemptions were slated to sunset within a few years. 

5 The exemptions surrounding hiring were over and above broader 
reforms aimed at improving teacher recruitment, retention, and evalu¬ 
ation, such as the open market transfer system (see Goertz, Loeb, & 
Wyckoff, 2011, for more detail on policies aimed at improving the 



teaching workforce during these years). Moreover, changes in state 
regulation surrounding teacher certification at the turn of the century 
meant that the pool of teachers competing for jobs was increasingly 
more qualified (at least in ways researchers are able to measure). 

6 Changes in the high school application process were introduced in 
academic year 2003-2004 via the High School Application Processing 
System (HSAPS). Under this system, all New York City (NYC) public 
school eighth graders are required to submit a ranked list of up to 12 
high schools citywide according to their personal preferences. The NYC 
Department of Education uses a computerized matching process to 
assign students to high schools based on their preferences, school selec¬ 
tion criteria, and seat availability (Abdulkadiroglu, Pathak, & Roth, 
2005, 2009). 

7 Howley, Strange, and Bickel (2000) observe that studies of out¬ 
comes recommend smaller school sizes than those based on inputs, and 
studies focusing on aspects of community in education recommend 
smaller sizes than those based on outcomes. 

8 Test scores are measured in z scores, which are standardized to 
have a mean of 0 and a standard deviation of 1 over all test takers in a 
grade and year. 

9 The Regents Examinations are a series of tests, aligned with 
New York State’s (NYS) Learning Standards, which New York students 
must pass in order to receive high school diplomas. They are designed 
and administered under the authority of the Board of Regents of the 
University of the State of New York (the state governing body for K-16 
education) and prepared by teacher examination committees and test¬ 
ing specialists. Examination scores range from 0% to 100%. To earn a 
Regents high school diploma, New York students need to obtain appro¬ 
priate credits in a number of specific subjects by passing yearlong or 
half-year courses, after which they must pass a Regents Examination 
in that subject area. This expectation is in addition to passing the 
courses themselves, the passing grade of which is based on an individual 
teacher’s or school’s own tests and class work. Starting with the cohort 
entering Grade 9 in 2001, and thus including our own cohorts, to 
receive a Regents high school diploma, students need to score a 65 or 
above in the following five content areas: Integrated Algebra (or Math 
A), Global History and Geography, U.S. History and Government, 
Comprehensive English, and any one science area. To earn an Advanced 
Regents diploma, students take additional credits in a foreign language, 
pass an additional Regents exam in science (at least one in life science 
and one in physical science), and pass a second Regents exam in math. 
Students in our cohorts also were allowed to graduate with local (not 
Regents) diplomas, which required passing any one of five Regents 
examinations with a score of at least 55%. The math exams offered 
for the cohorts in our study are Math A and Math B. Topics tested by 
the Math A Regents exam include equations and inequalities, prob¬ 
ability and statistics, and geometry. Math B, which is optional, is taken 
after the student has passed Math A. Topics that can be tested include 
concepts from trigonometry and advanced algebra as well as some pre¬ 
calculus and calculus. 

“’Students receiving a general equivalency diploma are not consid¬ 
ered graduates. We focus on the English and Math (Math A) Regents as 
these are the first exams required to be taken by all students before NYS 
graduation requirements began to change. 

1 ’Alternative high schools and high schools exclusively for special 
education students have different goals for graduation and testing than 
regular high schools. Specifically, rates of 4-year graduation and testing 
are expected to be lower. Although students can (and some do) travel 
outside Staten Island to attend another high school citywide, this is not 
common and very few travel outside Staten Island to attend a small 
high school. Moreover, there are no small high schools in Staten Island. 

12 Since we use cohort, rather than panel data, there is only one 
observation per student. 


13 A similar instrumental variables (IV) framework has been used in 
an educational evaluation of Chicago schools (Cullen, Jacob, & Levitt, 
2005), an evaluation of small schools in Chicago (Barrow, Clessens, & 
Schanzenbach, 2010), examinations of the effect of college attendance 
on earnings (Card, 1995) and on health behaviors (Currie & Morretti 
2003), and most recently, our evaluation of the small school reforms in 
NYC (Schwartz, Stiefel, & Wiswall [SSW], 2013). 

14 In our IV analysis, we allow the coefficients on the covariates to 
differ between the pre- and postreform cohorts to control for differences 
in how student characteristics affect performance may have changed over 
time The results are largely robust to using different functions of distances 
between schools. Including distance to nearest small and distance to nearest 
large school is essentially the same as using relative distance, for example. 

15 To some extent, this is a result of the closing of low-performing 
large schools. 

16 The covariates in later cohorts have statistically significant differ¬ 
ences in coefficients from ones in earlier years (p < .01). 

17 This finding is consistent with the finding in SSW (2013) that 
students in new small schools do better but not students in old small 
schools. 

18 We define schools that closed as schools serving students in 
cohort 2001 but not 2008. Schools that open are schools serving stu¬ 
dents in postreform cohorts but not in cohort 2001. 

19 See Department of Labor Statistics (2012). 

20 AU dollars are inflated to 2008 dollars using the Consumer Price 
Index, and districts are weighted by their enrollment to reflect differ¬ 
ences in size. 

21 Notably, these dollars do not include the support from founda¬ 
tions and philanthropies. Although private dollars compose less than 
1 % of the NYC Department of Education’s annual budget, they may 
provide flexibility to embark on reform efforts. The small schools reform 
in NYC was supported by $100 million in funding from the Gates 
Foundation, with additional funding from the Carnegie Corporation of 
New York and Open Society Institute. 

22 The only significant difference for the graduation outcomes was 
for the 2005 cohort. 

’’Common Core of Data Build a Table data tool: http://nces. 
ed.gov/ccd/bat. 
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